feat(sdk): 8 MiB python3 ingest guard + TypeScript SDK skill#116
Merged
Conversation
The python3 runtime (CPython WASM) reads sandbox files through an 8 MiB IPC bridge, so open() fails on larger files. Both SDKs now reject files over 8 MiB on ingest_files/ingestFiles with EFILE_TOO_LARGE_FOR_CPYTHON before sending. Pass allow_oversized/allowOversized to ingest anyway — the bytes stay usable from bash and js-exec, only python3 open() can't read them. Split into <8 MiB chunks to read large files from python3.
📝 WalkthroughWalkthroughClient-side validation now guards against CPython's 8 MiB file-read limit: Python and TypeScript SDKs raise ChangesCPython 8 MiB File-Size Guard for Ingest
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~22 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Owner
Author
|
@codex review |
check:version requires the top CHANGELOG heading to match package.json (0.3.0); the changeset file is the canonical record of unreleased changes, so version-locked dated sections are the repo convention.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 22c9abc0e6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The 8MB ingest guard lives entirely in clients/{python,typescript},
which are versioned independently of the sql-fs-api package the
changeset system tracks. A sql-fs-api bump would be inaccurate.
New skill plugins/sql-fs/skills/ts-sdk/ documents the TypeScript SDK
(sql-fs-sdk) with the same structure as py-sdk: SKILL.md, SETUP.md,
ref/{client,sandbox,models,errors}.md, and four runnable examples.
Adapted to the TS API surface — camelCase methods, options objects,
async SSE via for-await, try/finally + client.close() (no context
manager) — and documents the new ingestFiles allowOversized / 8 MiB
EFILE_TOO_LARGE_FOR_CPYTHON guard. Bumps the plugin to 1.1.1.
Root `biome check .` lints the plugin example .ts files; biome wraps the long collect() signature onto multiple lines.
E2E-testing the ts-sdk skill against a live local API surfaced that just-bash has no `uname` command (exit 127, command not found). The smoke-test / batch-probe examples used `uname -srm`, which would fail. Replaced with `pwd` (builtin) and `find ... | wc -l` across ts-sdk, py-sdk, and the api skill (whose endpoints.md also showed a fabricated "Linux ..." output).
Audited the py-sdk skill against clients/python/src/sqlfs/ and the new ts-sdk skill. Fixed: - 8 MiB python3 ingest guard (allow_oversized / EFILE_TOO_LARGE_FOR_CPYTHON) was only in ref/sandbox.md; now also in SKILL.md, ref/errors.md, ref/client.md - retry_on_5xx (a real exec/exec_batch param) was undocumented; added to ref/sandbox.md signatures + ref/client.md retry policy - stale plugins/sqlfs/ paths and /sqlfs: invocations corrected to the real plugins/sql-fs/ dir and /sql-fs: namespace
Mirror the py-sdk fix: the api SKILL.md referenced the non-existent plugins/sqlfs/ dir and /sqlfs: invocation. Corrected to plugins/sql-fs/ and /sql-fs:. The whole plugin is now free of stale path/namespace refs.
Hazzng
changed the title
Hazzng
added a commit
that referenced
this pull request
Jun 8, 2026
Bumps both SDK clients to 0.3.1 and adds CHANGELOG entries for the ingestFiles/ingest_files 8 MiB python3 guard (allowOversized / EFILE_TOO_LARGE_FOR_CPYTHON), whose code landed on main via #116. - TS: package.json + src/version.ts + CHANGELOG.md (check:version consistent) - Python: pyproject.toml + src/sqlfs/_version.py + CHANGELOG.md Merging this to main fires the SDK release workflows, publishing 0.3.1 to npm + PyPI with the guard code already present.
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two related strands, both about making the SQL-FS SDKs safe and well-documented to use:
ingestFiles/ingest_filesagainst files larger than 8 MiB that thepython3runtime can'topen().1. The 8 MiB ingest guard (Python + TS SDKs)
The
python3runtime (CPython compiled to WASM) reads sandbox files through an 8 MiB SharedArrayBuffer IPC bridge.open()on a larger file fails with an opaqueFileNotFoundError/OSError— the file appears to exist (os.listdirworks) but can't be read. A 17 MB CSV would ingest fine, then silently fail when a Python script tried to open it.Both SDKs now reject files > 8 MiB on ingest before anything is sent:
ValidationErrorcodeEFILE_TOO_LARGE_FOR_CPYTHON, client-side (statusundefined, no round-trip).allow_oversized/allowOversized— bytes stay usable from bash (cat/grep/awk) and js-exec; onlypython3 open()can't read them.2. New
ts-sdkskill + skill fixesplugins/sql-fs/skills/ts-sdk/— full operator guide for thesql-fs-sdkTypeScript package, mirroringpy-sdkfile-for-file (SKILL.md,SETUP.md,ref/{client,sandbox,models,errors}.md, 4 runnable examples). Adapted to the TS API (camelCase methods, options objects, async SSE viafor await,try/finally+client.close()).SKILL.md/ref/{client,errors,sandbox}.md; added the previously-undocumentedretry_on_5xxparam; fixed staleplugins/sqlfs/paths and/sqlfs:namespace.unamebug — e2e testing revealed just-bash has nouname(exit 127); replaceduname -srmwithpwd/find … | wc -lin ts-sdk, py-sdk, and the api skill (whoseendpoints.mdeven showed a fabricated"Linux ..."output).plugins/sqlfs///sqlfs:fix.1.1.0→1.1.1.No changeset: the change is confined to the SDK clients and the plugin skills, both versioned independently of the
sql-fs-apipackage the changeset system tracks.Test plan
pnpm typecheck+pnpm test(21 passing, incl. block + override cases)pytest(41 passing, incl. block + override cases),ruff,mypycleanpnpm lint(biome check .) clean across all 237 filescheck:versionconsistency check passesnpm i'dsql-fs-sdkpackage: auth bootstrap → create → ingest (guard blocks >8 MiB,allowOversizedbypasses, bash reads 9 MiB butpython3 open()fails) → exec / execBatch(readOnly) / execStream → readOnly violation → cleanup. All assertions passed.🤖 Generated with Claude Code